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ABSTRACT 

The notion that completeness of a goal intent will 
cause a fundamental change in the whole character of evaluation 
methodology is discussed. In addition, a series of methodological 
gaps are described. The focus was the lack of a methodic way to: (1) 

Establish pertinent goals; (2) Acquire goals such that the original 
intended outcome remain invariant throughout the evaluative process; 
(3) Ascertain whether the decision-maker uses the data provided by 
the evaluator; (4) Provide useful data for improvement and 
modification during the evaluative process; (5) Validate the 
measurability of a goal intent; (6) Train decision-makefcs as a 
participant in the evaluative process; and (7) Institutionalize a 
formal discipline for training educational evaluators. (Author/CK) 
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Ambiguities around the concept of evaluation combined with the 
things that the evaluator purports to do lead to a series of methodolo- 
gical gaps. The completeness aspect of a goal intent is one such gap. 
Completeness should be judged by the impact that evaluative data has on 
decisions made by decison-makers . 

Some Prevalent Methodological Gaps 

By methodological gap this writer means the absence of a set of 
procedures or methodology that enhances the understanding and mastery 
of an evaluation problem or issue. Since this definition is somewhat 
vague and lacks specificity, the number of existing methodological gaps 
might be quite large. In fact, the present state of the art might pos- 
sibly be that , to a great extent , methodological gaps have not yet been 
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articxxlated or publicized on any wide scale. 

In light of the foregoing, this writer will consider a limited 
number of methodological gaps only. They include the following: 

1. The lack of a systematic way of establishing pertinent goals 
of enterprise to be evaluated. For example, there is con- 
siderable methodology on how to formally state goals and a 
variety of taxonomies of goals (Bloom (1969) , Mager (1961), 
and McAshan (1970) discuss some), but almost no set of pro- 
cedures current in evaluation literature with respect to 
how to tease significant goals out of program literature or 
documents exist. Methodology for such activity is available- 
in other disciplines such as the natural and physical sciences 
and instructional technology and management. To a large 
extent goals are written such that they influence proposal 
funding. This overused purpose of a goal oftentimes obscures 
its real intent. 

2. The lack of the wherewithal to acquire a uniform set of 
goals that remain invariant throughout the evaluation of 
an enterprise irrespective of audience. There are times 
when goals should be changed during the in-progress stage 
of evaluation; e.g., a feature of formative evaluation. 

In fact, if the decision-maker decides to modify a goal 
on the basis of evaluative data, the evaluator, himself, 
may be useful resource provided the decison-maker recognizes 
that a change of goals implies a change of intended outcomes. 
For instance, it is one thing for a decision-maker to 
change a goal so that pertinent variables and their implica- 
tions are consistent with the already established intended 
outcomes. It is, indeed, another thing for decision-makers 
to change a goal so that variables and their implications 
are consistent with programmatic operations. This is 
especially true if such operations tend to produce outcomes 
that may differ markedly from the original intended outcomes 
of the goal. 

3. The lack of the wherewithal to ascertain whether the deci- 
sion-maker uses the data provided by the evaluator. The 
current notion of evaluation opts to provide data for 
decision-making by decison-makers . There is little evidence 
to support the fact that decision-makers themselves have 
either any influence over the data or affinity or respect 
for the data that is offered to aid them in their decision- 
making. This state of the art makes it conducive for the 
decision-maker to ignore such data. 
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The lack of a systematic way to provide useful data for 
improvement and modification during the in-progress stage 
of the evaluation. There are many reasons why there is no 
systematic way to provide useful data. Sometimes not enough 
useful data has been gathered; e.g. , if a laboratory test 
for biopsy analysis of possibly malignant tissue are in- 
accurate, that data alone is not helpful for systematic im- 
provement. Data is needed on processing the tissue from 
the moment of taking it from the patient through delivery 
to the laboratory, analysis of finding, and the report of 
results. There are other times when the way in which data 
are presented have little effect on whether such data are 
used for useful improvements in any systematic manner. 

There are many schools, for example, that take standardized 
achievement test data and simply store them in file cabinets 
or place them on the shelf. Too often evaluation data 
arrives too late to be of practical value to decis-on-makers . 
Evaluation data must provide a rational basis for modifica- 
tion should program operations warrant such change. That is, 
if (a) a deviation from the real goal intent of the decision- 
maker occurs and (b) a descrepancy with respect to the im- 
plementation of the decision-maker's real goal intent occurs, 
then, the evaluation data should provide implication for a 
systematic change that would lead to the attainment of the 
decision-maker's desired outcomes. 

5* The lack of the wherewithal to validate the measurability 
of a goal intent. Because of the multiplicity of the pur- 
pose of goals, their real intent seldom manifest themselves. 
In instances where goal intents are evident, too often they 
appear in a form that is either highly general or somewhat 
vague that needs to be delineated to the extent that they 
can be stated into specific and meaningful observable com- 
ponents . The fact that goals can be delineated to the ex- 
tent that they can be stated in the form of a behavioral 
component implies the following: 

a. The desired behavior can be observed directly 
from a qualitative perspective or; 

b. The desired behavior is measurable from a quanti- 
tative perspective. 

Thus giving rise to the development of either measurable 
techniques or observable techniques or both. Validity 
questions about these techniques reflects the inadequacy 
of evaluation methodology to date. 
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6. The lack of decision-maker training in relation to evalua- 
tion as a process . The degree to which most educational 
enterprises have clearly defined goals leaves something to 
be desired. This condition may explain why. it is entirely 
possible for public school evaluations to be meaningless. 

These evaluations reflect the confusion of administrators 
regarding educational programs which sometimes may be equal- 
ly meaningless. Due to the lack of theory and the absence 
of tradition of good practice it is also possible that some 
evaluators fail to fulfill evaluation needs, conceptually. 

There may be a relationship among the strength of educational 
programs, evaluation methodology, and the training provided 
in institutions for both decision-makers and evaluators. 

7* The lack of formal training for evaluators. Up to the 
present time practitioners in evaluation have received 
their training in evaluation techniques on a piecemeal 
basis. Some of their training was received in courses 
in tests and measurements; some of their training was ac- 
quired in educational statistics and psychology courses; 
and other parts of their training were tacked-on to methods 
courses. This piecemeal approach to the development of 
evaluation skills has generally been unsatisfactory. 

Educators of the field are normally poor evaluators , in part 
because of their inability to synthesize the varying frag- 
ments of training into functional skills. 

The term "methodology", with respect to this paper, means the 

\ 

science of methods. The methodological gaps discussed in the preceding 
paragraph amplifies the need for extensive research regarding evaluation 
methodology. In addition to these methodological gaps, there are others 
worthy of consideration. Among them is a very important one — the com- 
pleteness of a goal intent. A discussion of this gap is presented in 
the section which follows. 
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The Completeness of a Goal Intent 






The development of evaluation skills and methodology has not kept up 

with the increasing needs for evaluation. An important methodological 

concept from a practical point of view is comoleteness . By completeness, 

this writer means the attainment of the state such that there is a list 

of variables all of which meet the "non-fuzzy" concept cirterion and , at 

the same time, characterize the decision-maker's real goal intent fully. 

By "non- fuzzy" concept this writer means a variable with behavior that 

is either directly observable or measurable. Implicit* in the concept of 

completeness are focus and efficiency: These terms will be explained 

% 

later. Presumably, if goal statements are operationalized properly, then 
variables that can influence important decisions will be generated. 

Completeness connotes validity, which entails the question of whether 
an assessment technique serves its ultimate purpose adequately. In the 
case of evaluation which has the specific purpose of providing useful 
data for decision-making by deci son-makers , completeness is ascertained 
by the degree to which decision-makers use the data provided. The 
question of whether an assessment technique cerves its ultimate purpose 
adequately is determined by the amount of evaluative data used by the 
decision-maker. 
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For purposes of this paper, the notion of completeness pre- 
supposes the employment of the operationalisation of fuzzy concept 
(OFC) methodology to generate a viable list of variables. The OFC 
methodology— originated by Hutchinson (1969) — is a systematic way of 
delineating a somewhat vague statement into observable behaviors or 
states and providing a procedure for quantifying such behaviors . For 

a more detailed discussion interested persons are referred to Jones 

4 \ 

(in Press). 

In evaluation there are essentially two aspects of the complete- 
ness of a goal intent. They are (a) getting the decision-maker to 
articulate his real goal intent to the evaluator and (h) providing 
useful data for decision-making by decision-makers. 

In order to attain the completeness of a goal intent, the evaluator 
must elicit the decision-maker's entire goal intent. A way in which 
this can be accomplished is for the evaluator to lead the decision-maker 
through the appropriate phases of the OFC methodology. The main phases 
of the OFC methodology are: 

Procurement of goals 
Operationalization of goals 
Development of assessment techniques. 

Implementation of assessment techniques. 

Summarization of the evaluation— the report. 




6 



7 



An example situation of a somewhat vaguely stated goal for which the 

OFC methodology was applied, according to Jones (1970), is the goal 

( 

"Learn to work independently. Among the list of variables generated 
by the OFC methodology that characterized the decision-maker’s real in- 
tent were the following: 

1. Make a tentative schedule for studying at school. 

2. Make a tentative schedule for studying at home. 

3. Study according to the tentative schedule at home. 

4. Study according to the tentative schedule at home. 

5* Perform a science laboratory experiment independent of 
supervision. 

6. Make-up at least one goal similar to a teacher-made goal. 

7* Make-up at least one goal that is of interest to you. 

8. Devise ways of carrying out your goal(s). 

9. Do home-work assignments. 

10. Use reference books daily (dictionary and encyclopedia). 

There are times when evaluators enjoy the luxury of considering goals 
that are clearly stated. For instance, among the goals of the Guidance 
Counselor of the West Springfield, Massachusetts school system were the 
following : 

1. Be in class on time. » 

2. Pick up trash about your seats. 

# An administrator's goal for the pupil of a special section of the 
6th grade class in the West Springfield, Massachusetts public school 
district. 
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At a glance, it can be seen that it is indeed helpful for evaluators 
to work with clearly stated goals. This is particularly true if they are, 
in fact, behaviorally stated goals as in the case of the example cited. 

The evaluator can develop assessment techniques immediately. As a matter 
fact, the entire evaluative process is simplified by such luxury. • 

The fact that, for the most part, evaluators work with goals in 
the form of somewhat vague statements gives rise to a new role for evalua- • 
tors. That is, the role of clarifying goal statements so that they are 
- amenable to the evaluation of an enterprise. The OFC methodology is a 
way in which goal clarification can be attained anC the preservation of 
the real intent of the decision-maker's goal is guaranteed throughout 
the entire process. Therefore, it is reasonable to assume that a pertinent 
list of variables characterizing a goal intent can be generated. Moreover, 
it is reasonable to assume that variables comprising such a list have 
implications germane to an additional aspect of completeness — the concept 
of closure. 

The cyclic aspect of the OFC methodology — test of completeness— 
provides a guarantee that any arbitrary goal can actually be delineated 
into components such that is possible for alternatives to be exhausted. 

The matter of establishing closure via cyclic iterations is extremely : 
difficult and oftentimes impossible. Thus, giving rise to the inference 
that in general, evaluation as a process is incomplete. 
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The concept of closure means that all aspects of an arbitrary goal 
statement has been delineated with respect to decison-raaker relevancy. 

That is both consistency and completeness criteria have been attained. 

/ 

A main feature of the closure concept is that it provides the where- 
withal for any aspect of every goal statement to be accommodated. 

Now, as we turn to another aspect of completeness, let us con- 
sider the variable. For purposes of evaluation, a variable is defined 
as the smallest element of a goal intent. The variables alluded to 
earlier characterize what is known as an explicit variable. An explicit 
variable is an unambiguously stated manifestation of a goal intent that 
can be observed directly. Usually explicit variables manifest themselves 
in the form of simple sentences. The predicate of theVsentence dictates the 
evaluation activity. 

The evaluation activity suggests the data to be produced. That is, 
the data to be collected and analyzed. If these data are found to be 
useful for dec is on-making then implications about completeness with res- 
pect to evaluative results may be evident. On the contrary, however, if 
these data do not influence decision-making the converse holds. The 
amount of decisions made by the decision-maker without the use of evalua- 
tive data reflect the degree of incompleteness of the evaluation for 
some particular decision-maker. 

Included in the dpmain of completeness are the concepts of effi- 
ciency and focus, as stated earlier. Efficiency refers to the amount 
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of data actually used by the decision-maker in his deci son-making 
process. This implies that the amount of data not used by the decision- 
making process is the amount of inefficiency of the evaluation for that 
decision-maker. Focus refers to .the effectiveness of the evaluation 
design. Included in the purpose of the evaluation design is to faci- 
litate the collection and analysis of data to produce pertinent infor- 
mation for decison-makers. In particular, the focus of evaluation en- 
compasses the effectiveness of data as it stands in relation to the 
number of decisions made on the basis of the data provided. 

The concept of effectiveness includes the decision-maker's 
priorities »ri th respect to goal intent. That is, evaluative results 
should be such that data is provided for the important decisions and 
not provided for less important decisions. 

Summary 

The discussion of completeness was the most important feature of 
this paper. It is the belief of this writer that the notion of complete- 
ness of a goal intent will cause a fundamental change in the whole 
character of evaluation methodology. Further research on the complete- 
ness of a goal intent is in order. Once this concept is developed more 
fully so that its implications axe generalizable to other aspects of 
evaluation methodology, it should pave the way for the establishment 
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of* evaluation as a scientifically oriented discipline. 

Other features of this paper were a series of methodological 
gaps. The focus was the lack of a methodic way to: 

1. Establish pertinent goals. 

2. Acquire goals such that the original intended outcome remain 
invariant throughout the evaluative process. 

3. Ascertain whether the decision-maker uses the data provided 
by the evaluator. 

h. Provide useful data for improvement and modification during 
the evaluative process. 

5. Validate the measurability of a goal intent. 

6. Train decision-makers as a participant in the evaluative process. 

7. Institutionalize a formal discipline for training educational 
evaluators . 
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